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ABSTRACT 

In this paper, we study the performance of IPOP- s *aACM- 
ES and BIPOP-"*aACM-ES, recently proposed self-adaptive 
surrogate-assisted Covariance Matrix Adaptation Evolution 
Strategies. Both algorithms were tested using restarts till a 
total number of function evaluations of 10 6 D was reached, 
where D is the dimension of the function search space. We 
compared surrogate-assisted algorithms with their surrogate- 
less versions IPOP-aCMA-ES and BIPOP-CMA-ES, two al- 
gorithms with one of the best overall performance observed 
during the BBOB-2009 and BBOB-2010. 

The comparison shows that the surrogate-assisted ver- 
sions outperform the original CMA-ES algorithms by a fac- 
tor from 2 to 4 on 8 out of 24 noiseless benchmark problems, 
showing the best results among all algorithms of the BBOB- 
2009 and BBOB-2010 on Ellipsoid, Discus, Bent Cigar, Sharp 
Ridge and Sum of different powers functions. 

Categories and Subject Descriptors 

G.1.6 [Numerical Analysis]: Optimization — global opti- 
mization, unconstrained optimization; F.2.1 [Analysis of 
Algorithms and Problem Complexity]: Numerical Al- 
gorithms and Problems 

General Terms 

Algorithms 

Keywords 

Benchmarking, black-box optimization, evolution strategy, 
CMA-ES, self-adaptation, surrogate models, ranking sup- 
port vector machine, surrogate-assisted optimization 

1. INTRODUCTION 

When dealing with expensive optimization objectives, the 
surrogate-assisted approaches proceed by learning a surro- 
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gate model of the objective, and using this surrogate to re- 
duce the number of computations of the objective function 
in various ways. 

Many surrogate modelling approaches have been used within 
Evolution Strategies (ESs) and Covariance Matrix Adapta- 
tion Evolution Strategy (CMA-ES): Radial Basis Functions 
network [9], Gaussian Processes [18], Artificial Neural Net- 
work [3], Support Vector Regression [13], Local- Weighted 
Regression [121 IT], Ranking Support Vector Machine (Rank- 
ing SVM) |17l 1141 [TO] . In most cases, the surrogate model 
is used as a filter (to select Xp re promising pre-children) 
and/or to estimate the fitness of some individuals in the cur- 
rent population. An example of surrogate-assisted CMA-ES 
with filtering strategy can be found in |14] , 

A well-known drawback of surrogate-assisted optimization 
is a strong dependence of the results on hyper-parameters 
used to build the surrogate model. Some optimal settings 
of hyper-parameters for a specific set of problems can be 
found by offline tuning, however for a new problem they are 
unknown in the black-box scenario. Moreover, the optimal 
hyper-parameters may dynamically change during the opti- 
mization of the function. 

Motivated by this open issues, new self-adapted surrogate- 
assisted s *aACM-ES algorithm have been proposed combin- 
ing surrogate-assisted optimization of the expensive func- 
tion and online optimization of the surrogate model hyper- 
parameters [15] . 

2. THE ALGORITHMS 

2.1 The ( M / Mu , ( A)-CMA-ES 

In each iteration t, (/i//i„ , A)-CMA-ES 7 samples A new 
solutions Xi G R 15 , where i = 1, . . . , A, and selects the best 
\i among them. These \i points update the distribution of 
parameters of the algorithm to increase the probability of 
successful steps in iteration t + 1. The sampling is defined 
by a multi-variate normal distribution, A/^ra*, a' 2 C'), with 
current mean of distribution m t , D x D covariance matrix 
C' and step-size <r'. 

The active version of the CMA-ES proposed in [8] [11] in- 
troduces a weighted negative update of the covariance ma- 
trix taking into account the information about A — fi worst 
points as well as about /x best ones. The new version im- 
proves CMA-ES on 9 out of 12 tested unimodal functions by 
a factor up to 2, and the advantages are more pronounced 
in larger dimension. While the new update scheme does not 
guarantee the positive-definiteness of the covariance matrix, 



it can be numerically controlled [8]. Since in our study we 
do not observe any negative effects of this issue, we will use 
aCMA-ES, the active version of the CMA-ES, for compari- 
son with the surrogate-assisted algorithms. 

2.2 The "ACM ES 

The S *ACM-ES [T5J is the surrogate-assisted version of 
the (fj,/fi w , A)-CMA-ES, where the surrogate model is used 
periodically instead of the expensive function for direct op- 
timization. The use of Ranking SVM allows to preserve 
the property of CMA-ES of invariance with respect to rank- 
preserving transformation of the fitness function. The prop- 
erty of invariance with respect to the orthogonal transforma- 
tion of the search space is preserved thanks to the definition 
of the kernel function by the covariance matrix, adapted 
during the search. 

In S *ACM-ES we perform the following surrogate-assisted 
optimization loop: we optimize the surrogate model / for h 
generations by the CMA-ES, then we continue and optimize 
the expensive function f(x) for one generation. To adjust 
the number of generations h for the next time, the model 
error can be computed as a fraction of incorrectly predicted 
comparison relations that we observe, when we compare the 
ranking of the last A evaluated points according to f(x) and 
/. The S *ACM-ES uses the generation of the CMA-ES as 
a black-box procedure, and it has been shown in [T5J, that 
the improvement of the CMA-ES from passive to active ver- 
sion (aCMA-ES) leads to a comparable improvement of its 
surrogate-assisted versions ("ACM-ES and s *aACM-ES). 

The main novelty of the S *ACM-ES is the online optimiza- 
tion of the surrogate model hyper-parameters during the op- 
timization of the fitness function. The algorithm performs 
the search in the space of model hyper-parameters, generat- 
ing Xhyp surrogate models in each iteration. The fitness of 
the model can be measured as a prediction error of the rank- 
ing on the last A evaluated points. This allows the user to 
define only the range of hyper-parameters and let algorithm 
to find the most suitable values for the current iteration t. 

The detailed description of S *ACM-ES is given in [TS]. 

2.3 The Benchmarked Algorithms 

For benchmarking we consider four CMA-ES algorithms 
in restart scenario: IPOP-aCMA-ES [S], BIPOP-CMA-ES 
[U, IPOP- s *aACM-ES and BIPOP- s *aACM-ES[T5]. For IPOP- 
3 *aACM-ES and BIPOP- s *aACM-ES we use the same pa- 
rameters of the CMA-ES and termination criteria in IPOP 
and BIPOP scenario as in the original papers. The default 
parameters for "ACM-ES algorithms are given in |15j . 

3. RESULTS 

Results from experiments according to [5j on the bench- 
mark functions given in [2j [6] are presented in Figures [2] [3] 
and[3]and in Tables [JJ and O The expected running time 
(ERT), used in the figures and table, depends on a given 
target function value, ft = / op t + A/, and is computed over 
all relevant trials (on the first 15 instances) as the number 
of function evaluations executed during each trial while the 
best function value did not reach ft, summed over all trials 
and divided by the number of trials that actually reached ft 
[5J[T5]. Statistical significance is tested with the rank- 
sum test for a given target A/t (10 -8 as in Figure [2]) using, 
for each trial, either the number of needed function eval- 
uations to reach A/ t (inverted and multiplied by —1), or, 



if the target was not reached, the best A/-value achieved, 
measured only up to the smallest number of overall function 
evaluations for any unsuccessful trial under consideration. 

The IPOP- s *aACM-ES and BIPOP- s *aACM-ES represent 
the same algorithm ( s *aACM-ES) before the first restart oc- 
curs, therefore, the results are very similar for the uni-modal 
functions, where the optimum usually can be found without 
restarts. The s *aACM-ES outperforms aCMA-ES usually by 
a factor from 2 to 4 on /i ,/2 ,/s o ,/i l ,/i 2 ,/i 3 and / 14 for 
dimensions between 5 and 20. The speedup in dimension 2 
is less pronounced for problems, where the running time is 
too short to improve the search. This is the case for /g Lin- 
ear Slope function, where the speedup can be observed only 
for dimension 20, because the optimum can be found after 
about 200 function evaluations. To improve the search on 
functions with small budgets it would make sense to use the 
surrogate model right after the first (g a tart = 1) generation 
of the CMA-ES, while in this study this parameter g start 
was set to 10 generations. 

The good results on uni-modal functions can be explained 
by the fact, that while using the same amount of informa- 
tion (all previously evaluated points), s *aACM-ES processes 
this information in a more efficient way by constructing the 
approximation model of the function. Similar effect of more 
efficient exploitation of the available information can be ob- 
served for aCMA-ES in comparison to CMA-ES. 

The speedup on multi-modal functions is less pronounced, 
because they are more difficult to approximate and the final 
surrogate model often has a bad precision. In this case the 
adaptation of the number of generations leads to an oscilla- 
tion of h close to 0, such that the surrogate model is not used 
for optimization or used for small number of generations. 

The BIPOP versions of CMA-ES usually perform better 
than IPOP on /23 and /24, where the optimum is more likely 
to be found if use small initial step-size. This leads to over- 
all better performance of the BIPOP versions and BIPOP- s * 
aACM-ES in particular. The better performance of the lat- 
ter in comparison with BIPOP-CMA-ES can be partially 
explained by the fact of using the active covariance matrix 
update. However, this is not the case for /20 — /24 functions 
in 5-D and /15-19 in 20-D (see Fig. [3]and Fig. g}. 

The 3 *aACM-ES algorithms improve the records in dimen- 
sion 10 and 20 on /7,/io,/n,/i2,/i3,/i4,/i5,/i6,/20. 

4. CPU TIMING EXPERIMENT 

For the timing experiment the IPOP- s *aACM-ES was run 
on f\, /g, /10 and /15 without self-adaptation of surrogate 
model hyper-parameters. The crucial hyper-parameter for 
CPU time measurements, the number of training points was 
set Ntraining = [40 + 4D 1,7 J as a function of dimension D. 

These experiments have been conducted on a single core 
with 2.4 GHz under Windows XP using Matlab R2006a. 

On uni-modal functions the time complexity of surrogate 
model learning increases cubically in the search space di- 
mension (see Fig. [JJ and quadratically in the number of 
training points. For small dimensions (D < 10) the overall 
time complexity increases super-linearly in the dimension. 
The time complexity per function evaluation depends on 
the population size, because one model is used to estimate 
the ranking of all points of the population. This leads to a 
smaller computational complexity on multi-modal functions, 
e.g. /15 Rastrigin function, where the population becomes 
much larger after several restarts. 
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Figure 1: CPU cost per function evaluation of 
IPOP-aACM-ES with fixed hyper-parameters. 

The results presented here does not take into account the 
model hyper-parameters optimization, where Xh VP surrogate 
models should be build at each iteration, which leads to an 
increase of CPU time per function evaluation by a factor of 
\ hyp . For BIPOP- s *aACM-ES and IPOP- s *aACM-ES X hyp 
was set to 20. 

5. CONCLUSION 

In this paper, we have compared the recently proposed 
self-adaptive surrogate-assisted BIPOP- s *aACM-ES and IPOP- 
3 *aACM-ES with the BIPOP-CMA-ES and IPOP-aCMA- 
ES. The surrogate-assisted s *aACM-ES algorithms outper- 
form the original ones by a factor from 2 to 4 on uni-modal 
functions, and usually perform not worse on multi-modal 
functions. The s *aACM-ES algorithms improve the records 
on 8 out of 24 functions in dimension 10 and 20. 
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Figure 2: Expected running time (ERT in number of /-evaluations) divided by dimension for target function 
value 10 -8 as log 10 values versus dimension. Different symbols correspond to different algorithms given in 
the legend of /i and /24. Light symbols give the maximum number of function evaluations from the longest 
trial divided by dimension. Horizontal lines give linear scaling, slanted dotted lines give quadratic scaling. 
Black stars indicate statistically better result compared to all other algorithms with p < 0.01 and Bonferroni 
correction number of dimensions (six). Legend: o: BIPOP-CMA, V: BIPOP-saACM, *: IPOP-aCMA, □: 
IPOP-saACM. 
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Figure 3: Bootstrapped empirical cumulative distribution of the number of objective function evaluations 
divided by dimension (FEvals/D) for 50 targets in lO^ 8 " 2 ' for all functions and subgroups in 5-D. The "best 
2009" line corresponds to the best ERT observed during BBOB 2009 for each single target. 
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Figure 4: Bootstrapped empirical cumulative distribution of the number of objective function evaluations 
divided by dimension (FEvals/D) for 50 targets in 10'~ 8 " 2 ' for all functions and subgroups in 20-D. The "best 
2009" line corresponds to the best ERT observed during BBOB 2009 for each single target. 
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A/ opt 


lcl 


IcO 


lc-1 


lc-3 


lc-5 


lc-7 


#succ 


f6 


114 


214 


281 


580 


1038 


1332 


1 ") / 1 ") 


BIPOP-C 


2.3(1) 


2.1(0.6) 


2.2(0.6) 


1.7(0.2) 


1.3(0.3) 


1.3(0.2) 


15/15 


BIPOP-s 


1.9(0.7) 


1.8(0.6) 


2.1(0.9) 


1.8(0.9) 


1.3(0.5) 


1.4(0.5) 


15/15 


IPOP-aC 


2.5(0.8) 


2.1(0.6) 


2.2(0.4) 


1.6(0.2) 


1.2(0. 1) 


1.2(0. 1) 


15/15 


IPOP-sa 


2.3(1) 


2.1(0.9) 


2.6(1) 


1.9(0.8) 


1.5(0.5) 


1.7(0.7) 


15/15 


A/ opt 


lcl 


leO 


lc-1 


lc-3 


lc-5 


lc-7 


#succ 


f7 


24 


324 


1171 


1572 


1572 






BIPOP-C 


5.0(5) 


1.5(1) 


1(1) 


1(0.9) 


1(0.9) 


1 (0.9) 


15/15 


BIPOP-s 


4.5(3) 


1.4(1) 


0.83(0.5) 


0.91(0.7) 


0.91(0.7) 


0.93(0.7) 


15/15 


IPOP-aC 


4.0(3) 


0.87(0.2) 


0.70(0.6) 


0.69(0.5) 


0.69(0.5) 


0.70(0.5) 


15/15 


IPOP-sa 


3.7(2) 


1.2(1) 


0.68(0.5) 


0.77(0.9) 


0.77(0.9) 


0.90(0.9) 


15/15 


A/ pt 


lcl 


IcO 


lc-1 


le-3 


lc-5 


le-7 


#succ 


f8 


73 


273 


336 


391 


410 


422 


15/15 


BIPOP-C 


3.2(1) 


3.7(2) 


4.5(2) 


4.8(2) 


5.1(2) 


5.4(2) 


15/15 


BIPOP-s 


2.2(0.6) 


2.5(2) 


2.5(2) 


2.5(2) 


2.5(1) 


2.5(1) 


15/15 


IPOP-aC 


2.8(1) 


3.0(1) 


3.6(1) 


4.0(1) 


4.2(1) 


4.5(1.0) 


15/15 


IPOP-sa 


2.3(1) 


1.9(0.7) 


2.0(0.6) 


2.0(0.5) 


2.0(0.5) 


2.1(0.5) 


15/15 


A/opt 


lcl 


IcO 


lc-1 


lc-3 


lc-5 


lc-7 


#succ 


f9 


35 


127 


214 


300 


335 


369 


15/15 


BIPOP-C 


5.8(2) 


8.7(3) 


7.2(2) 


6.4(2) 


6.3(1) 


6.2(1) 


15/15 


BIPOP-s 


4.2(1) 


4.1(4) 


3.1(2) 


2.6(2) 


2.5(2) 


2.3(1) 


15/15 


IPOP-aC 


5.4(1) 


6.2(2) 


5.7(1) 


5.0(1.0) 


5.0(0.8) 


4.9(0.8) 


15/15 


IPOP-sa 


5.0(2) 


3.1(1) 


2.5(0.6) 


2.1(0.5) 


2.0(0.5) 


2.0(0.4) 


15/15 


A/opt 


lcl 


IcO 


lc-1 


lc-3 


lc-5 


lc-7 


#succ 


flO 


349 


500 


574 


626 


829 


880 


15/15 


BIPOP-C 


3.5(0.8) 


2.9(0.4) 


2.7(0.4) 


2.8(0.2) 


2.3(0.2) 


2.4(0.1) 


15/15 


BIPOP-s 


0.76(0.1) 


0.59(0.1) 


0.58(0.1) , 3 0. 60(0.1) , 3 0. 51(0.1) , 53(0.1) |, 


15/15 


IPOP-aC 


2.5(0.8) 


2.2(0.3) 


2.1(0.3) 


2.2(0.3) 


1.8(0.2) 


1.9(0.2) 


15/15 


IPOP-sa 


0.77(0.2) 


0.61(0.1) 


0.57(0.1) ^30.60(0. 1) ^ 3 0. 5 l(0.1)j_ 


3 0. 53(0.1) i3 


15/15 


A/opt 


lcl 


leO 


lc-1 


lc-3 


lc-5 


le-7 


#succ 


fll 


143 


202 


763 


1177 


1467 


1673 


15/15 


BIPOP-C 


8.4(3) 


7.2(2) 


2.2(0.3) 


1.6(0.2) 


1.4(0.1) 


1.3(0.1) 


15/15 


BIPOP-s 


2.1(0.3) 


1.6(0.2) 


0.47(0.1) , 4 0. 34(0.0) | 4 0. 30(0.0) , 4 0. 29(0.0) 


15/15 


IPOP-aC 


5.6(0.6) 


4.7(0.5) 


1.4(0.1) 


1.0(0.1) 


0.95(0.1) 


0.92(0.1) 


15/15 


IPOP-sa 


2.0(0.4) 


1.6(0.3) 


0.46(0.1)^40.34(0.1)^40.30(0.1)^40.29(0.0)^4 


15/15 


A/opt 


lcl 


leO 


le-1 


lc-3 


lc-5 


lc-7 


#succ 


fl2 


108 


268 


371 


461 


1303 


1494 


15/15 


BIPOP-C 


11(12) 


7.4(8) 


7.4(6) 


7.7(5) 


3.3(2) 


3.3(2) 


15/15 


BIPOP-s 


2.6(0.8) 


2.1(2) 


2.1(2) 


2.5(3) 


1.4(1) 


1.7(2) 


15/15 


IPOP-aC 


8.8(7) 


5.9(7) 


5.7(5) 


6.0(5) 


2.6(2) 


2.6(2) 


15/15 


IPOP-sa 


3.8(3) 


2.9(2) 


2.8(2) 


2.8(2) 


1.2(0.9) 


1.2(0.9) 


15/15 



A/opt 
fl3 

BIPOP-C 
BIPOP-s 
IPOP-aC 
IPOP-sa 



A/ 



fl4 

BIPOP-C 
BIPOP-s 
IPOP-aC 
IPOP-sa 

A /opt 



fl5 

BIPOP-C 
BIPOP-s 
IPOP-aC 
IPOP-sa 



A/ c 



fl6 

BIPOP-C 
BIPOP-s 
IPOP-aC 
IPOP-sa 



A/, 



fl7 

BIPOP-C 
BIPOP-s 
IPOP-aC 
IPOP-sa 



A/c 



5.2 

3.4(3) 

3.8(6) 
4.3(5) 
4.9(4) 
lcl 



fl8 

BIPOP-C1(0 
BIPOP-s 
IPOP-aC 
IPOP-sa 



A/ 



fl9 

BIPOP-C 
BIPOP-s 
IPOP-aC 
IPOP-sa 



A/ 



f20 

BIPOP-C 
BIPOP-s 
IPOP-aC 
IPOP-sa 



A/ 



f21 

BIPOP-C 
BIPOP-s 
IPOP-aC 
IPOP-sa 



A/ 



f22 

BIPOP-C 
BIPOP-s 
IPOP-aC 
IPOP-sa 

A /opt 



f23 

BIPOP-C 
BIPOP-s 
IPOP-aC 
IPOP-sa 



A/ 



121- 



f24 

BIPOP-C 
BIPOP-s 
IPOP-aC 
IPOP-sa 



132 

3.9(3) 

1.2(0.3) 

3.0(2) 

1.2(0.4) 

lcl 



IcO 

5.4(3) 

1.3(0.4) 

4.1(2) 

1.1(0.3) 

IcO 



lc-1 
~~250 

5.9(3) 
1.3(0.4) 
4.2(0.8) 
1.2(0.3) 

lc-1 



lc-3 



lc-5 



lc-7 



1310 1752 2255 

1.6(0.3) 1.5(0.2) 1.7(0.8) 
0.37(0. 1)^ 4 0. 34(0. 1)^.32(0.1)^ 
1.2(0.2) 1.2(0.1) 1.1(0.1) 
0.33(0.1)^ 4 0.35(0.1)^ 4 0.33(0 

lc-3 lc-5 lc-7 



#succ 
15/15 
15/15 
15/15 
15/15 
l) i4 |l5/15 
#succ 



10 

1.1(1.0) 

1.5(2) 
1.5(2) 
2.8(2) 
lcl 



2.8(1) 

2.3(0.6) 

2.2(1) 

2.5(0.6) 

IcO 



58 

3.7(0.9) 
2.4(0.5) 
3.2(0.8) 
2.5(0.6) 
lc-1 



139 

4.6(0.7) 
2.2(0.3) 
3.6(0.5) 
2.1(0.3) 
lc-3 



251 

5.4(0.5) 
1.8(0.2) 
3.8(0.6) 
1.8(0.2) 
lc-5 



476 

4.5(0.3) 
1.3(0.1) 
2.9(0.3) 
1.3(0.1) 
lc-7 



15/15 
15/15 
15/15 
15/15 
15/15 
#succ 



1.6(2) 
1.4(1) 
1.5(2) 
1.6(1) 
lcl 



9310 19369 
1.5(1) 1.2(0.7) 
0.59(0.5) 0.65(0.5) 
0.89(0.5) 1.0(0.7) 
0.42(0.4)^ 0.72(0.6) 

IcO lc-1 



20073 

1.2(0.7) 

0.64(0.5) 

1.0(0.7) 

0.71(0.6) 

lc-3 



20769 

1.2(0.7) 

0.62(0.5) 

1.0(0.7) 

0.70(0.6) 

lc-5 



21359 

1.2(0.7) 

0.61(0.4) 

1.0(0.7) 
0.69(0.6) 

lc-7 



14/15 
15/15 
15/15 
15/15 
15/15 
#succ 



120 612 2662 10449 11644 12095 

3.0(3) 3.6(3) 2.6(1) 1.3(2) 1-4(2) 1.4(2) 

3.2(4) 4.3(2) 16(1) 0.82(0.7) 0.77(0.7) 0.79(0.7) 

3.9(4) 2.4(2) 17(2) 0.82(0.7) 0.84(0.6) 0.85(0.6) 

3.1(4) 2.4(2) 1-0(1) 0.52(0.6)^ 0.55(0.6)^ 0.55(0.5)^ 

lcl IcO lc-1 lc-3 lc-5 lc-7 



15/15 
15/15 
15/15 
15/15 
15/15 

#s 



215 899 

1(0.2) 1(2) 

1.0(1) 1-2(2) 

0.89(0.4) 0.53(0.2) 

1.8(0.4) 1.1(1) 
IcO lc-1 



3669 
1(0.7) 
0.92(0.5) 
0.77(0.5) 

0.85(0.5) 
lc-3 



6351 

1(0.5) 

1.1(0.4) 

1.00(0.5) 

1.2(0.5) 

lc-5 



7934 

1.2(0.5) 

1.4(1) 

1.1(0.9) 

1.4(0.8) 

lc-7 



15/15 
15/15 
15/15 
15/15 
15/15 
#succ 



103 

7) 

2.5(1) 
3.5(0.8) 
2.8(0.8) 



lcl 



378 
3.4(5) 
3.1(4) 
1.6(0.5) 
1.5(0.5) 
IcO 



3968 
1(1) 

0.82(1.0) 
0.70(0.5) 
0.64(0.7) 
lc-1 



9280 

1(0.3) 

0.76(0.7) 

0.77(0.3) 
0.85(0.7) 
lc-3 



10905 
1.2(0.7) 
1.2(0.7) 
0.80(0.3) 
1.0(0.4) 
lc-5 



12469 
1.3(0.6) 
1.4(0.7) 
0.84(0.3) 
1.1(0.4) 
lc-7 



1 1 242 1.2c5 1.2c5 1.2c5 

20(16) 2801(5070) 161(175) 1(0.7) 1(0.7) 1(0.7) 

16(14) 1834(1246) 02(142) 0.85(0.6) 0.85(0.6) 0.85(0.6) 

14(10) 1207(1125) 123(152) 0.95(0.7) 0.96(0.7) 0.96(0.7) 

19(16) 1931(1477) 250(254) 1.4(1) 1.4(1) 1.4(1) 

lcl IcO lc-1 lc-3 lc-5 lc-7 



16 

3.3(3) 
3.1(2) 
3.9(2) 
2.4(2) 
lcl 



851 
8.2(10) 
3.9(4) 

10(4) 
3.5(3) 
IcO 



38111 

2.8(1) 
1.7(1) 
1.4(2) 
1.6(1) 
lc-1 



54470 
2.1(0.8) 
1.2(1.0) 
1.1(1) 
1.2(1) 
lc-3 



54861 

2.2(0.8) 

1.2(1.0) 

1.1(1) 

1.2(1) 

lc-5 



55313 

2.2(0.8) 

1.2(1.0) 

1.1(1) 

1.2(1) 

lc-7 



2.3(2) 
1.9(1) 
3.5(1) 
2.9(2) 
lcl 



1157 

14(9) 
3.4(4) 
7.3(8) 
1.6(2) 
IcO 



1674 
24(35) 
4.9(8) 

32(36) 
28(23) 
lc-1 



1705 
25(36) 
5.0(8) 

33(40) 
28(26) 
lc-3 



1729 
25(36) 

4.9(8) 
33(41) 
28(26) 
lc-5 



1757 
25(36) 

4.9(8) 
33(41) 
27(26) 
lc-7 



71 

6.9(11) 

2.2(4) 

8.8(10) 

3.4(5) 

lcl 



386 
20(14) 

3.2(4) 
21(26) 
12(13) 

IcO 



938 
45(94) 
14(27) 
65(74) 
85(258) 
lc-1 



1008 
42(88) 
13(25) 
270(385) 
116(267) 
lc-3 



1040 

41(85) 
13(24) 
262(368) 
113(258) 



1068 
40(83) 
13(23) 
257(351) 
110(252) 
lc-7 



3.0 
1.7(2) 
4.2(4) 
1.6(1) 

lcl 



518 
13(15) 
14(8) 
20(19) 
13(17) 
IcO 



14249 
3.7(4) 
1.1(1) 

76(89) 



31654 
1.8(2) 
0.91(0.9) 

34(56) 



33030 
1.8(2) 
0.88(0.8) 

33(38) 



34256 

1.8(2) 

0.97(0.8) 
32(51) 



(18 



le-7 



1622 

2-1(2) 

1.8(1) 

2.6(2) 

1.9(1) 



2.2c5 
1.6(3) 
1.0(1) 

41(46) 
42(55) 



6.4c6 9.6c6 

1(1.0) 1(1.0) 

0.69(0.8) 0.97(1) 

oo oo 

11(13) oo 



1.3c7 1.3e7 

1(1) 1(1) 

0.73(0.8) 0.73(0.8) 

oo oo le6 

oo co 5e6 



15/15 
15/15 
15/15 
15/15 
15/15 
#succ 



15/15 
15/15 
15/15 
15/15 
15/15 
#succ 



14/15 
15/15 
15/15 
15/15 
15/15 
#succ 



14/15 
15/15 
15/15 
14/15 
15/15 
#succ 



14/15 
15/15 
15/15 
9/15 
15/15 



15/15 
15/15 
15/15 
8/15 
15/15 
#succ 



3/15 
3/15 
6/15 
0/15 
0/15 



Table 1: Expected running time (ERT in number of function evaluations) divided by the respective best ERT 
measured during BBOB-2009 (given in the respective first row) for different Af values in dimension 5. The 
central 80% range divided by two is given in braces. The median number of conducted function evaluations 
is additionally given in italics, if ERT(10~ 7 ) = oo. #succ is the number of trials that reached the final target 
/opt + 1CP 8 . Best results are printed in bold. 
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ti 

BIPOP-C 
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IPOP-aC 
IPOP-sa 



A/< 



EL. 



9(2) 14(3) 

4.0(0.2) 5.1(0.4) 

7.9(1) 14(1) 

3.9(0.2) 5.0(0.4) 

lcl IcO 



f2 

BIPOP-C 
BIPOP-s 
IPOP-aC 
IPOP-sa 

A/ opt 
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35(7) 

6.8(1) 
22(3) 
7-3(1) 

lcl 



f3 
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BIPOP-s 
IPOP-aC 
IPOP-sa 

A/ op t 
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10(7) 

11(15) 
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BIPOP-s 
IPOP-aC 
IPOP-sa 



fS 

BIPOP-C 
BIPOP-s 
IPOP-aC 
IPOP-sa 



A/q 



pt 



41 

5.1(0.8) 
4.7(0.7) 

1(1) 
4.5(0.7) 

lcl 



BIPOP-C 
BIPOP-s 
IPOP-aC 
IPOP-sa 



A/ Q 



IT 

BIPOP-C 
BIPOP-s 
IPOP-aC 
IPOP-sa 

A/ppt 



f8 

BIPOP-C 
BIPOP-s 
IPOP-aC 
IPOP-sa 

A/opt 



f9 
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BIPOP-s 
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IPOP-sa 

A/ppt 
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IPOP-aC 
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IPOP-sa 
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43 



43 

20(2) 

6.6(0.7) 
20(1) 

6.5(0.5) 

lc-1 



43 

33(4) 

10(0.7) 

33(2) 

9.5(0.7) 

le-3 



43 

45(3) 
13(0.8) 
45(2) 
13(0.7) 
le-5 



43 

57(3) 

16(1) 
58(2) 
16(0.8) 

lc-7 



15/15 
15/15 
15/15 
15/15 
15/15 
#succ 



386 
40(4) 

8.0(1) 
27(2) 

8.3(2) 
IcO 



387 
44(4) 
8.9(1) 

29(1) 
8.9(2) 
lc-1 



390 
47(2) 
10(1) 
31(2) 
10(2) 
lc-3 



391 
48(2) 
10(1) 
33(2) 
10(1) 
lc-5 



393 
50(2) 
10(1) 
34(2) 
10(1) 
lc-7 



7626 



7635 



76 13 



7646 
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1722 



1.9e4(2e4) oo 
lcl IcO 
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oo 
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oo 
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lc-7 
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oo 


oo 
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OO 


oo 2e7 


oo 
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OO 
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9/15 

0/15 

0/5 

0/15 

0/5 

#succ 



41 

6.2(1) 
5.3(0.7) 
6.2(0.9) 
5.1(0.7) 
IcO 



41 

6.3(1) 
5.4(0.7) 
6-2(1) 
5.2(0.8) 
lc-1 



41 

6.3(1) 
5.4(0.7) 
6-2(1) 
5.2(0.8) 
lc-3 



41 

6.3(1) 
5.4(0.7) 
6-2(1) 
5.2(0.8) 

lc-5 



41 

6.3(1) 
5.4(0.7) 
6.2(1) 
5.2(0.8) 
lc-7 



1296 

1.5(0.4) 

1.4(0.3) 

1.6(0.3) 

1.5(0.4) 

lcl 



2343 

1.3(0.2) 

1.2(0.2) 

1.3(0.2) 

1.2(0.3) 

IcO 



3413 

1.2(0.2) 

1.1(0.2) 

1.1(0.2) 

1.1(0.2) 



5220 

1.1(0.2) 

1.1(0.2) 

1.1(0.1) 

1.1(0.2) 

lc-3 



6728 

1.2(0.1) 

1.3(0.3) 

1.1(0.1) 

1.2(0.2) 

lc-5 



8409 

1.2(0.1) 

1.4(0.3) 

1.1(0.1) 

1.3(0.3) 

lc-7 



1351 

1(0.5) 

1.0(0.9) 

1.6(2) 

1.0(1) 

lcl 



4274 

4.9(2) 

1.6(0.6) 

2.7(1) 

1.6(0.6) 

IcO 



9503 

3.5(0.6) 

0.84(0.3) 

1.6(0.5) 

0.92(0.6) 

lc-1 



16524 16524 16969 

2.2(0.3) 2.2(0.3) 2.1(0.3) 
0.61 (0.1)^. 61(0.1)^. 60(0.1)^ 
0.99(0.3) 0.99(0.3) 1.0(0.3) 
0. 66(0. 3)^ 2 0.66(0.3)^ 2 0.65(0.3) 12 



-3 



lc 



le-7 



2039 

1.0(1) 

1.3(0.2) 

3.5(0.8) 

1.4(0.2) 

lcl 



3871 

4.0(0.7) 

1.5(0.9) 

3.5(0.5) 

1.3(0.1) 

IcO 



4040 

4.3(0.6) 

1.5(0.9) 

3.7(0.6) 

1.4(0.1) 

lc-1 



4219 

4.5(0.6) 

1.6(0.8) 

3.9(0.6) 

1.4(0.1) 

lc-3 



4371 

4.6(0.6) 
1.6(0.8) 
3.9(0.5) 
1.4(0.1) 

lc-5 



4484 

4.6(0.6) 
1.6(0.8) 
4.0(0.5) 
1.4(0.1) 

lc-7 



1716 

4.7(2) 

1.5(0.3) 

4.1(0.7) 

1.6(0.4) 

lcl 



3102 

5.7(1) 

1.7(0.2) 

4.6(0.5) 

1.8(1) 

IcO 



3277 

6.0(1) 

1.7(0.2) 

4.9(0.5) 
1.9(1) 

lc-1 



3455 

6-1(1) 

1.8(0.2) 

5.0(0.5) 
1.9(1) 

lc-3 



3594 

6.1(1.0) 

1.8(0.2) 

5.0(0.5) 
1.9(1.0) 
lc-5 



3727 

6.1(0.9) 

1.7(0.2) 

5.0(0.5) 
1.9(1.0) 
lc-7 



7413 8661 10735 14920 17073 17476 

1.9(0.2) 1.8(0.2) 1.6(0.1) 1.2(0.0) 1.1(0.0) 1.1(0.0) 
0. 36(0. 1)^ 4 0. 35 (0.0) 440.31(0.0)^40. 24(0. 0)^ 4 0. 23 (0.0)^ 4 0. 23(0.0 
1.2(0.2) 1.2(0.1) 1.0(0.1) 0. 80(0. 0)^ 4 0. 73(0. 0)^ 4 0. 75(0 " 
0.35(0.1)^40.36(0.1)^40.31(0.0)^40.24(0.0)^40.22(0.0)^40.23(0.0)^, 



lcl 



leO 



lc-1 



le-3 



le-5 



le-7 



1002 

5) 

2.5(0.4) 
4.5(0.2) 
2.5(0.5) 
lcl 



2228 
5.1(0.3) 
1.2(0.2) 
2.3(0.1) 
1.2(0.2) 
IcO 



6278 9762 12285 14831 15/15 

1.9(0.1) 1.4(0.0) 1.2(0.0) 1.0(0.0) 15/15 

0.44(0.1) 0.30(0.0)^40.26(0.0)^40.23(0.0)^. 15/15 

0.87(0.1) 0. 64(0. 0)^ 4 0. 56(0. 0)j_ 4 0. 50(0. 0)j_ < 15/15 

0.45(0.1) 0. 31(0. 1)440. 26(0. 1)^ 4 0. 23(0. 0)4^ 15/15 



lc-1 



le-3 



le-5 



le-7 



1042 1938 

3.0(2) 4.0(3) 

0.99(0.9) 1.1(1) 

2.6(1) 3.0(2) 

0.67(0.1) 0.88(0.7) 



13827 
2.0(0.7) 



2740 4140 12407 

4.5(3) 4.5(2) 1.9(0.' , 

1.2(0.9) 1.2(0.9) 0.55(0.3)^ 0.59(0.3)^ 

3.2(2) 3.1(1) 1.3(0.4) 1.4(0.4) 

1.1(0.7) 1.2(0.5) 0. 56(0. 2)^ 2 0. 60(0. 



A/ c 



EL. 



15/15 
/15 
/15 
15/15 
15/15 
#succ 



15/15 
0/15 
0/5 
0/15 
0/5 

#succ 



15/15 
15/15 
15/15 
15/15 
15/15 

#succ 



15/15 
15/15 
15/15 
15/15 
15/15 

#succ 



15/15 
15/15 
15/15 
15/15 
15/15 
#succ 



15/15 
15/15 
) i4 |15/15 
0) 44 15/15 
' 15/15 
#succ 



15/15 
15/15 
15/15 
15/15 
2)^ 2 15/15 



fl3 

BIPOP-C 
BIPOP-s 
IPOP-aC 
IPOP-sa 

A/c 

Tit" 

BIPOP-C3 
BIPOP-s 
IPOP-aC 
IPOP-sa 



15/15 
15/15 
15/15 
15/15 
15/15 
#aucc 
15/15 
15/15 
15/15 
15/15 
15/15 
#succ 



A/c 



EL. 



9(1) 
3.2(1) 
3.6(0.8) 
3.0(0.6) 
lcl 



fl5 

BIPOP-C 
BIPOP-s 
IPOP-aC 
IPOP-sa 



A/ 



30378 1.5c5 3.1c5 3.2c5 4.5e5 4.6e5 15/15 

1(0.4) 2.0(0.8) 1.4(0.5) 1.4(0.5) 1(0.3) 1(0.3) 15/15 

0.65(0.6) 1.3(0.6) 0.91(0.7) 0.89(0.6) 0.66(0.5) 0.65(0.5) 15/15 

0.82(0.3) 1.1(0.3) 0.71(0.2) 0.72(0.2) 0. 53(0. 2)430. 54(0. 2)^ 2 15/15 

0.60(0.5) 0.92(0.7) 0.53(0.4) 0.52(0.4) 0.37(0.3)^40.37(0.3)^15/15 

lcl IcO lc-1 lc-3 lc-5 lc-7 #succ 



fl6 

BIPOP-C 
BIPOP-s 
IPOP-aC 
IPOP-sa 



A/ 



fl7 

BIPOP-C 
BIPOP-6 
IPOP-aC 
IPOP-sa 

A/ Q£ t_ 



FT8 

BIPOP-C 
BIPOP-s 
IPOP-aC 
IPOP-sa 



A/ 



fl9 

BIPOP-C 
BIPOP-s 
IPOP-aC 
IPOP-sa 



A/ 



EL. 



f20 

BIPOP-C 
BIPOP-s 
IPOP-aC 
IPOP-sa 

A /opt 



f21 

BIPOP-C 
BIPOP-s 
IPOP-aC 
IPOP-sa 

A /opt 



f22 

BIPOP-C 
BIPOP-s 
IPOP-aC 
IPOP-sa 



A/ 



EL. 



#succ 



f23 

BIPOP-C 
BIPOP-s 
IPOP-aC 
IPOP-sa 

A /opt 



f24 

BIPOP-C 
BIPOP-s 
IPOP-aC 
IPOP-sa 



652 
4.3(6) 
1.1(0.9) 
3.6(3) 
1.7(2) 

lcl 



IcO 

2021 

2.7(2) 

0.89(0.7) 

3.4(3) 

1.7(0.8) 

IcO 



le-1 

2751 

5.1(6) 

1.4(1.0) 

3.7(2) 

1.5(0.7) 

lc-1 



lc-3 



lc 



le 



18749 24455 30201 

1.5(0.8) 2.3(2) 3.0(2) 
0. 38(0. 1)4,40. 42(0. 2) i4 0. 40(0.1)^ 
0.80(0.4) 1.3(0.7) 1.3(0.7) 
O.34(0.2)^40.37(0.1)^ 4 0.41(0.2 

lc-3 lc-5 lc-7 



239 

2.9(0.4) 
1.8(0.6) 
2.7(0.3) 
1.8(0.3) 

IcO 



304 

3.7(0.4) 
1.9(0.4) 
3.5(0.3) 
1.9(0.4) 
lc-1 



932 

4.1(0.3) 
1.5(0.2) 
3.2(0.3) 
1.4(0.2) 

lc-3 



1648 

6.2(0.5) 

1.4(0.2) 

3.9(0.2) 

1.4(0.1) 

lc-5 



15661 

1.2(0.1) 
0.23(0.0)^ 
0.67(0.1 
0.23(0 

lc-7 



___n__ 

15/15 
15/15 
15/15 
15/15 

#succ 
15/15 
15/15 
15/15 



) j.415/15 
0)^4|l5/15 



1384 
1.7(0.4) 

1.9(0.6) 

2.8(5) 

1.8(0.8) 

lcl 



27265 

1.0(0.7) 

0.74(0.4) 

1.1(0.5) 

0.55(0.4) 

IcO 



77015 

1.2(0.7) 

0.51(0.3) 

0.88(0.7) 

0.77(0.8) 

lc-1 



1.9c5 

1(0.7) 
0.60(0.5) 
0.80(0.5) 
0.52(0.4) 

lc-3 



2.0c5 

1(0.7) 

0.84(0.5) 

0.82(0.5) 

0.55(0.4) 

lc-5 



2.2c5 

1(0.7) 

0.83(0.5) 

0.76(0.5) 

0.50(0.3)^ 

lc-7 



15/15 
15/15 
15/15 
15/15 
15/15 
#succ 



63 

2.2(2) 

3.2(2) 
2.3(2) 
2.5(2) 
lcl 



1030 

1(0.3) 

1.2(0.4) 

0.89(0.2) 

0.91(0.3) 

IcO 



4005 

1(1) 

2.7(3) 

0.50(0.1) 

0.98(1) 

lc-1 



30677 
1.2(1) 
1.2(0.7) 
0.82(0.3) 

1.2(0.5) 
lc-3 



56288 

1.3(0.6) 

1.2(0.5) 

0.83(0.5) 

1.2(0.4) 

lc-5 



80472 

1.4(0.7) 

1.4(0.8) 

0.87(0.3) 

1.1(0.5) 

lc-7 



621 

1.0(0.4) 

1.0(0.3) 

1.2(0.4) 

0.96(0.5) 

lcl 



3972 
2.4(2) 
1.5(1) 
1.5(2) 
1.4(2) 
IcO 



19561 
1.2(0.9) 
0.92(0.4) 
0.75(0.4) 
0.91(0.6) 
lc-1 



15/15 
15/15 
15/15 
15/15 
15/15 
#succ 



67569 
1.1(0.6) 
0.96(0.4) 
0.91(0.4) 
0.78(0.5) 
lc-3 



1.3c5 
1.7(0.7) 
1.6(0.6) 
0.78(0.4) 
0.88(0.4) 
lc-5 



1.5c5 
1.6(0.6) 
1.6(0.5) 
0.83(0.3) 
1.3(0.8) 
lc-7 



169(74) 
143(52) 
166(60) 
154(50) 
lcl 



1 3.4c5 
2.4e4(lc4)1.2(0.6) 
2.5e4(le4)0.42(0.3) 
2.9c4(2e4)0.63(0.4) 
3.0c4(2c4)0.61(0.5) 
IcO lc-1 



6.2c6 6.7c6 6.7c6 15/15 

1(0.3) 1(0.3) 1(0.3) 15/15 

0.72(0.4)^ 0.73(0.4) 0.73(0.4) 15/15 
0.43(0.2)^30.44(0.2)^30.44(0.2)^.15/15 
0.33(0.1) 0.32(0.2) 0.32(0.2) ]l4/15 
lc-3 lc-5 lc-7 #succ 



82 

4.3(1) 

2.9(0.5) 

4.7(1) 

2.8(0.5) 

lcl 



46150 
9.2(4) 
2.1(1) 
3.2(1) 
1.7(0.8 

IcO 



3.1e6 5.5e6 5.6c6 

1(0.5) 1(0.3) 1(0.3) 

0.97(0.7) 0.87(0.4) 0.86(0.4) 
0.83(0.4) 0.58(0.2) 0.59(0.2) 
0.49(0.2)^30.45(0.2)4^ 0.45(0.2)^ 



le-1 



le-3 



le-5 



5.6e6 

1(0.3) 

0.85(0.4) 

0.60(0.2) 

0.45(0.2)^ 

lc-7 



561 

3.2(6) 

2.6(4) 

1.9(4) 

2.6(4) 

lcl 



6541 
55(48) 

1.5(1) 
81(109) 
53(94) 
IcO 



14103 
48(86) 

6.0(11) 
66(94) 
157(308) 
lc-1 



14643 
46(84) 

5.8(11) 
64(88) 
151(297) 
le-3 



15567 
43(84) 

5.5(10) 
60(85) 
142(279) 
le-5 



17589 
39(77) 
4.8(9) 
54(77) 
126(247) 
le-7 



467 

6.8(13) 
7.7(9) 

462(1351) 

175(98) 

lcl 



5580 
13(21) 
100(96) 
264(443) 
978(1807) 
IcO 



23491 24948 
215(267) 202(247) 
178(320) 173(301) 



26847 

188(227) 

168(274) 



1.3e5 

37(48) 

35(54) 

00 lc6 



3.2 

4.3(5) 
3.0(6) 
4.1(6) 
4.3(6) 
lcl 



1614 

32(33) 
21(13) 
2.3e4(3e4) 
2.9e4(3e4) 
IcO 



67457 4.9e5 

1(0.8) 2.0(1) 

0.61(0.3) 1.4(1) 

906(1022) co 



8.1e5 

1.2(0.9) 

1.3(1) 



8.4e5 

1.2(0.9) 

1.3(1) 



le-7 



1.3c6 
1(0.9) 
0.99(1) 

25(32) 
28(30) 



7.5e6 
1(0.9) 
0.88(0.7) 

4.5(5) 
38(43) 



5.2e7 
1(1.0) 
0.80(0.! 



5.2e7 
1(1) 

0.80(0.! 



5.2e7 
1(1) 

0.80(0.8) 



5.2e7 
1(1.0) 
0.79(0.8 

00 5e6 
cc 2c't 



15/15 
15/15 
15/15 
15/15 
15/15 
#succ 



14/15 
14/15 
15/15 
15/15 
15/15 

#succ 



15/15 
13/15 
15/15 
9/15 
15/15 
#succ 



12/15 
5/15 

15/15 
0/15 
0/15 

#BUCC 



15/15 

15/15 

15/15 
0/15 
0/15 

#succ 

3/15 

3/15 

6/15 

0/15 

0/15 



Table 2: Expected running time (ERT in number of function evaluations) divided by the respective best ERT 
measured during BBOB-2009 (given in the respective first row) for different A/ values in dimension 20. The 
central 80% range divided by two is given in braces. The median number of conducted function evaluations 
is additionally given in italics, if ERT(1CP 7 ) = oo. #succ is the number of trials that reached the final target 
/opt + 10~ 8 . Best results are printed in bold. 



